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ABSTRACT 

This paper proposes both theoretical and empirical approaches to differentiating the community of inquiry 
(Col) in online classes at individual and group levels. Following the example of research on 
organizational climate, it assesses the strength of shared perceptions of teaching presence, social presence, 
and cognitive presence. The paper develops a theory of composition that relates isomorphic constructs of 
these presences at the individual and class levels. Hypotheses are made about the agreement among 
students on the survey that is used to measure individual perceptions of the three presences of the Col. 
These are tested using a set of statistics designed to measure the extent to which shared perceptions of 
these presences exist within classes. Strong evidence of shared perceptions is found, and potential 
practical and theoretical applications are described. 
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I. INTRODUCTION 

The concept of a community of inquiry (Col) was developed to describe phenomena that occur among all 
the members of online classes—in other words, to describe their interactions as a community. According 
to this view, “Deep and meaningful learning ... takes place in a community of inquiry composed of 
instructors and learners as the key participants in the educational process. The model assumes that in this 
community, learning occurs through the interaction of three core components: cognitive presence, 
teaching presence, and social presence” [1, p. 51-52]. Research on the Col measures the presences both at 
the group level and at the individual level. Years of research, conducted by coding and counting 
discussion posts from entire classes, have validated the theory [1, 2, 3, 4], Recently, there has been an 
explosion of research on the Col that measures individual perceptions of teaching, social, and cognitive 
presences in online classes using surveys [5, 6], Measuring the Col through individual survey responses 
creates some degree of confusion about the level of analysis of the concept: Is the Col framework 
founded on shared perceptions about a learning community, or does it refer to the individual perceptions 
and behaviors of students who happen to be taking a class? If Col truly represents a shared learning 
community, should the model also incoiporate the extent to which these perceptions or behaviors are 
shared? 

A multilevel approach can help to address some recent critiques of the Col framework. Rourke and 
Kanuka argued that there was little evidence that social and teaching presence lead to “deep, meaningful 
learning” [7]. Annand further questioned the role of social presence, particularly the aspects that reflect 
interaction among students; he argued that “social presence does not impact cognitive presence in a 
meaningful way” [8, p. 52] and held that individual differences between students are ignored in the Col 
paradigm. Xin approached the model from the opposite perspective, holding that all online 
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communication is social in nature, and described the interconnectedness of social, teaching, and cognitive 
presence in determining the learning environment [9]. These critiques focus attention on the social aspect 
of the Col, inviting examination of the level of analysis: How much of online learning is individual (and 
therefore should the Col describe what individual students perceive and do), and how much is a group 
phenomenon (and therefore should the Col describe what all the members of the group jointly perceive 
and do)? The reality may involve two parallel constructs, one focusing on the student and one on the 
learning community or class. 

This study examines the Col from the perspective of level of analysis. The research on organizational 
climate—in which individual perceptions of organizations, if shared, are aggregated to reflect group or 
organizational-level phenomena—is summarized. Both the theoretical and data analytic approaches to 
organizational climate are used as templates; a theory of composition is proposed in which social 
presence, teaching presence, and cognitive presences are described at both the level of the individual 
student and the class, and it is argued that shared perceptions of the class environment are a critical 
feature of a true Col in online classes. A set of analytical techniques used to determine the strength of 
group shared perceptions are adapted to test hypotheses about the strength of class-level community; this 
approach can be used to measure group-level Col by calculating the agreement among individual students 
on responses to the Col survey. 

II. ORGANIZATIONAL CLIMATE AND COMPOSITION THEORY 

Organizational researchers have studied climate in organizations for more than fifty years [10, 11, 12, 13, 
14]. Two isomorphic concepts have been identified and examined at different levels of analysis: 
psychological climate and organizational climate. Psychological climate is “the individual’s cognitive 
representation” of salient aspects of his/her situation and circumstances that mediates his or her 
perception of the environment, focusing attention and guiding interpretations of what occurs [15]. This 
representation is affected by the nature of the environment as well as the individual’s prior experiences, 
beliefs, needs, goals, and competencies [13, 15]. The psychological climate creates a frame through which 
individuals interpret the environment, and this frame affects what actions they take [13, 15, 16], which in 
turn affects the environment in a reciprocal process [15]. 

Organizational climate is the shared perception of what the work environment is like [17], or “how we do 
things around here” [18]. There are many objects, or referents, of the perceptions that make up the 
psychological and organizational climate. Some researchers have identified a set of categories that all 
employees use to frame the experience of work, such as the four factors of leader support and facilitation; 
role stress and lack of harmony; job challenge and autonomy; and workgroup cooperation, warmth and 
friendliness [13, 19]; or climate for learning, performance, and avoiding failure [20]. Other researchers 
focus on climate for a specific type of behavior or concern, such climate for service [17, 18], technical 
updating [21], safety [14, 22, 23], procedural justice [24], and ethical behavior and policies [25, 26], 
Regardless of the referent, organizational climate affects how group members perceive their work and 
environment, and the climate affects member behaviors, such as following safety rules, updating technical 
skills, and providing high (or low) levels of service [18, 21, 23]. Organizational climate also affects 
outcomes such as job satisfaction and absenteeism [27]. Organizational climate can be examined at 
different levels of aggregation, such as work teams, departments, or an entire organization, and can differ 
across levels, such that a team can have one climate but the department in which it exists can have another 
[22, 28], 

Organizational climate is affected both by structural factors of the organization, such as the size of groups 
and human resource practices, and by the cognitions and actions of members [13, 18, 27, 29, 30], The 
formation of organizational climate is “interactive and reciprocal. That is, similar individuals are ... 
exposed to similar features within contexts, and share their interpretations with others in the setting. Over 
time this process tends to result in consensus on climate perceptions” [31, pp. 546-547]. Several studies 
have found that groups that have greater degrees of interaction and greater interdependence on work tasks 
have stronger agreement on the climate [24, 32]. Newcomers are socialized into the shared climate, 
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although individuals also affect the organizational climate by their behaviors, which reflect their 
experiences, values and perceptions [24, 29]. 

Many researchers have found that group leaders play a critical role in developing both psychological and 
organizational climate [14, 18, 20, 27, 31]. According to Dragoni, “[Readers model the behavior they 
deem appropriate, provide direct and indirect feedback on whether group members have met expectations, 
and reward individuals who exhibit expected behaviors” [20, p. 1086]. The closeness of the relationship 
employees have with leaders affects the climate; employees who have closer relationships with their 
leaders have more positive views of the climate and greater consensus about it [31]. Similarly, groups that 
have transformational leaders, who are proactive and engaged closely with subordinates, have more 
consensus on climate than do groups with nontransformational leaders [14]. 

The construct of climate in organizations has much in common with the Col in classrooms. Both describe 
perceptions of the environment shared among members and leaders; both describe psychological spaces 
that are created by the interaction between structural factors and participants, and in which group 
members and leaders (i.e., teachers) reciprocally affect and are affected by the shared space. Both refer to 
what happens in the group; in the case of the Col, the context is a group created for the puipose of 
learning. 

A. Composition Theories 

The concepts of individual and organizational climate are differentiated from each other in composition 
theories, which guide understanding of related phenomena that exist at different levels. These kinds of 
theories “specify the functional relationships among phenomena or constructs at different levels of 
analysis (e.g., individual level, team level, organizational level) that reference essentially the same content 
but that are qualitatively different at different levels” [33, p. 234], According to Chan’s [33] widely cited 
analysis, there are five different kinds of composition theories: additive, direct consensus, referent-shift 
consensus, dispersion, and process theories. In additive theories, the group-level concept reflects the sum 
or average of the concept at lower levels, such as the research productivity of a university department. In 
direct consensus theories, the group-level concept exists only when members at lower levels of 
aggregation demonstrate consensus, or agree with one another. For example, if employees of a hank 
branch share a common understanding of how they should treat customers, there can be an identifiable 
climate for sendee within the branch. That common understanding held at the group level is different 
from the perceptions of each individual, because it creates a norm that affects new employees, who 
change their behavior to match it. In order to aggregate individual responses in a direct consensus theory, 
there must be demonstrable agreement among the members. Once that is established, the individual 
perceptions can be summed or averaged to create a measure of the group’s climate. 

Referent-shift consensus theories are similar in that they require consensus among group members, but 
the referent concept changes from the individual to group level; for example, perceptions of self-efficacy 
refer to individuals, while perceptions of team efficacy reflect shared beliefs about a team’s ability to 
accomplish a task. Dispersion theories link concepts at different levels by viewing the degree of 
agreement or disagreement among members of the group as the concept of interest. Last, process 
composition theories describe how phenomena at different levels develop over time, rather than 
describing a static state. Organizational climate has been studied using direct consensus, referent-shift, 
and dispersion theories; all have implications for measurement. 

B. Measuring Organizational Climate 

There is a significant degree of agreement about how to measure organizational climate: individuals are 
asked to describe the climate, and those responses are averaged, summed, or otherwise combined for a 
group or larger unit if sufficient agreement exists [11, 27, 33, 34, 35]. In a study guided by a direct 
consensus theory, organizational climate is examined using questions that refer to individual behaviors, 
beliefs, and values (e.g., “The supervisor to whom 1 report is cold to me”). The individual responses are 
then integrated into a shared perception of climate [14, 17, 22, 23, 30, 32, 33, 36, 37]. 
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Several different analytical indices have been recommended to assess the degree of agreement among 
group members, and different approaches have been recommended to determine what constitutes a 
sufficient level of agreement. Details of the different statistics, and their advantages and disadvantages, 
are presented in a Technical Appendix. The statistics r WG and r W G(j) [35], Average Deviation (AD), 
1CC(1) and 1CC(2) are very commonly used to measure within-group homogeneity and between-group 
variance, and in recent years many researchers have included multiple indices within their studies of 
workplace climate [14, 17, 22, 23, 32, 36, 37]; this practice is recommended for determining the extent to 
which group perceptions are shared [34, 39]. 

The next section will develop a direct consensus composition theory for the Col, linking individual and 
group-level teaching, social, and cognitive presences. The Col bears many similarities to the construct of 
organizational climate in that both describe a shared set of perceptions of the environment and behavioral 
expectations in a group (i.e., class section) that is structured and guided by a leader. 

III. A COMPOSITION THEORY OF THE COMMUNITY OF INQUIRY 

The Col model takes a “collaborative constructivist perspective on the teaching and learning transaction 
... [where] collaboration is seen as an essential aspect of cognitive development since cognition cannot be 
separated from the social context” [40, p. 92], According to this perspective, knowledge is created 
through communication and interaction. It is intrinsically tied to the social context of the class; learning in 
online courses occurs through the interaction of the faculty, student, and course materials, mediated by 
technology. Deep, higher-order learning occurs in, and as a result of, discourse that takes place among the 
members of a class or other learning community [41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52], The 
community of inquiry model that describes learning processes in online courses is intrinsically a group 
phenomenon, generally existing within an online class and comprised of the students and teacher(s) of 
that class. 

There have been two main approaches to measuring the Col elements: (1) coding and counting transcripts 
of asynchronous discussions, which focus on the group level, and (2) conducting surveys of students, 
which focus on the individual level. In most studies that use transcript coding, each sentence or message 
is coded, and the codes are aggregated across all members of the group who participated in the discussion 
by summing, averaging, or other using some other calculation [1, 4, 49, 53, 54, 55, 56, 57, 58, 59, 60, 61, 
62, 63, 64, 65], Most of the research aggregates posts to examine the community of inquiry that exists 
within a class, although some look at larger levels, such as disciplinary areas [66] and some at learning 
groups within a class [55], 

Coding and counting discussion threads is painstaking and labor-intensive, so an alternative approach to 
examining the Col was developed: surveys filled out by students. The Community of Inquiry Survey has 
been extensively validated in a wide range of universities with very large samples in two countries [5, 6, 
46, 47, 56, 67, 68, 69, 70, 71]. The instrument consists of 34 items measuring social presence, cognitive 
presence, and teaching presence. Individual students respond to questions describing their perception of 
multiple indicators of each presence. 

Survey responses have generally been analyzed using the student as the level of analysis. For example, 
research has examined the relationships among different presences, and their correlation with individual 
attitudes, such as course satisfaction [46, 47, 51, 52, 56, 65, 67, 72, 73], Other research examined 
relationships between course factors (such as design, discipline, or class size) and individual student 
scores on Col surveys [67, 68]. In a few cases, individual survey responses have been aggregated into 
means to create class-level variables [e.g., 53, 54, 63]; however, no assessment of the agreement among 
class members was undertaken. Overall, the complexities of the level of analysis have not been addressed. 


The following sections describe each element of the Col framework at both the individual and class 
section (group) levels. 
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A. Social Presence 

Social presence describes the strength of the social relationships and emotional connection among the 
members of a class or learning community. It has three elements: identifying with a learning community, 
communicating openly in a trusting environment, and developing interpersonal relationships [44, 74]. 
This construct has evolved over the past twelve years, moving from an emphasis on immediacy, 
projecting a sense of “self’ into the digital space, and developing emotional connection and open 
communication [1, 62, 72, 75] to focus more on educational goals and behaviors. 

Some of the changes in the framing of these concepts over time may reflect the transition from the 
original methodology of measuring group phenomena through aggregated discussion posts [1, 4] to the 
more recent approach of measuring the Col through individual survey responses [e.g., 5, 6], For example, 
the original element of affective expression became framed as developing interpersonal relationships', the 
total amount of affect expressed in a group discussion forum is now viewed as a set of actions that 
individuals take to develop relationships. Similarly, the facet of group cohesion became identifying with 
the community', groups have cohesiveness, while individuals identify with groups. 

Social presence reflects public behaviors on the part of individual students and faculty that are perceived 
by others and jointly form a collective perception of people in the group and of how “things are done” in 
the learning community. Student actions to relate with others in the learning community create a cohesive 
group when most students engage in those actions. In this formulation, individual-level social presence 
refers to the feelings of an individual and the actions that an individual takes to become a part of a 
community of inquiry, including personally identifying with the community, communicating in a trusting 
and comfortable environment, and developing interpersonal relationships with others in the class. 
Community-level social presence refers to the aggregate or shared sense of belonging that the students 
experience, the overall comfort and trust experience of the learning group, and the interpersonal 
relationships that exist among all members. 

Many student actions are contributions to discourse made in public posts or in live sessions, available to 
be viewed by others. This visibility should lead to agreement among class members about the level of 
social presence in the class. However, students vary in their interaction with learning groups due to 
personal factors, such as self-efficacy [59, 71], amount of prior online experience [67], and circumstances 
such as family and work demands, which can limit time for active participation [76]. Students range 
widely in their engagement with their peers through reading or responding; those who are not involved in 
creating a shared community tend to perceive less connection with others. Swan and Shih found that 
individual behavior is related to perceptions of social presence; students who perceived the most social 
presence displayed nearly 50% more indicators of social presence in their own discussion posts than did 
students who perceived the least social presence [65]. 

This phenomenon occurs in workplace climate as well, where interaction with others affects the levels of 
climate perceived even when there is a high degree of agreement [27]. Although everyone does not 
participate equally in a community, it is nonetheless a community. While individual levels of social 
presence may vary, there is a measurable community-level social presence in online classes that are 
designed and taught to support it; as Arbaugh and Hwang note, “a community will not be created unless 
participants actually engage each other” [77, p. 12], 

Teachers affect the degree of social presence by the way that they design assignments, such as using 
group activities, as well as by teaching activities such as creating informal discussion areas, rewarding 
students for having discussions with one another, modeling openness and encouraging it among students, 
and through many other teaching behaviors [44, 50, 56, 65, 78, 79], Therefore, this theory also holds that 
there are differences between classes in the degree of community-level social presence that exists in each 
class. Because course design, instructor behavior, and student characteristics all may affect social 
presence, sections of the same course are likely to differ from one another. Therefore the focus on 
agreement and disagreement is placed at the level of a class section rather than at the level of a unique 
course. 
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This is a direct consensus composition theory in that it proposes agreement among class members 
regarding answers to questions about each person’s personal perceptions, feelings, and experiences in the 
online class. If there is strong agreement among the students about these perceptions, feelings, and 
experiences, a community with social presence would exist. There would also be an expectation that 
classes would differ from each other in the extent to which they exhibit community-level social presence. 
There would then be a justification for considering the effects of community-level social presence in 
online learning research. 

Hypothesis 1 a: It is hypothesized that there will be strong agreement among members of a class section 
on perceptions of social presence. 

Hypothesis lb: It is hypothesized that class sections will differ in their community-level social presence. 

B. Teaching Presence 

Teaching presence is the extent to which the course instructor designs learning experiences, guides and 
directs student work, and facilitates interaction to support deep learning [4, 44, 46, 47, 50, 73, 77]. Akyol 
and Garrison referred to it as the provision of “leadership throughout the course of study” [54, p. 235]. 
This concept has expanded from its original focus on the design and support of discourse in asynchronous 
discussions to include the design of all learning activities and the provision of feedback [47, 77, 79, 80]. 
While instructor behavior is critical, student perceptions of that behavior are just as important in 
developing a community of inquiry. For example, an instructor who intends to provide clear guidance but 
who is perceived as condescending or overly directive will not be effective in creating an environment 
that supports exploration and deep learning. Similarly, if instructors provide information that students 
cannot access (perhaps due to difficulty locating it), it will not help students learn. Therefore, teaching 
presence should be measured through the perceptions of students. 

Teaching behavior is akin to group leadership; just as organizational leaders set goals, prioritize tasks, 
give feedback, and manage day-to-day work for employees, instructors set learning goals, structure 
activities, solve problems, facilitate work, provide information, and give feedback. Teachers lead the class 
in the achievement of its goal, which is learning. As with organizational leaders, most actions taken by 
teachers occur in public venues; these actions include the design of course elements, learning activities, 
and discussion prompts; the communication of evaluative criteria and rubrics; public announcements to 
guide movement through the course and address administrative issues; and posting in the discussions to 
support extended learning. The only actions that are not shared throughout the class are individual e-mails 
and private feedback on work. Therefore, perceptions of teaching presence are likely to be widely shared. 
In this theory of composition, individual-level teaching presence refers to each student’s perception of the 
instructor’s actions to support teaching by designing learning experiences, guiding learning, and 
providing feedback and direct instruction. Community-level teaching presence, on the other hand, is the 
shared perception of the instructor’s actions to support teaching by designing learning materials and 
activities and organizing the class’s progression through them; facilitating discourse among the members 
in a range of modalities (in discussions, groups, webinars, or other settings); and providing feedback and 
direct instruction for the entire group. 

If there is a strong shared agreement among the students on these perceptions, a community with teaching 
presence would exist. Teaching presence is affected by the individual instructor, so class sections would 
also differ in their community-level teaching presence scores. This would justify considering the effects 
of community-level teaching presence in online learning research. 

Hypothesis 2a: It is hypothesized that there will be strong agreement between members of a class section 
on perceptions of teaching presence. 

Hypothesis 2b: It is hypothesized that class sections will differ in their community-level teaching 
presence. 
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C. Cognitive Presence 

Cognitive presence has been defined as “the extent to which the participants in ... a community of inquiry 
are able to construct meaning through sustained communication” [40, p. 89]. It reflects the extent to 
which students engage in all four stages of the practical inquiry process [81], including addressing 
triggering problems, exploring the concepts and issues that underlie the problems, integrating information 
to identify solutions, and evaluating those solutions and considering further applications [3]. 

One might imagine that cognitive presence is more of an individual than a shared phenomenon; because 
learning occurs within each student’s mind, cognitive presence might be viewed as an individual and 
unique procession through the stages of deep learning. However, in a community of inquiry these stages 
result from students interacting with the learning activities created by the faculty (teaching presence), 
under faculty guidance (teaching presence), and are supported by interaction with other students (social 
presence) [82], Several studies have found that teaching and social presence together predict cognitive 
presence [46, 56, 71, 83], Thus, like social and teaching presence, cognitive presence has both an 
individual and a community component. Individual-level cognitive presence is viewed as the extent to 
which a specific student progresses through the four stages of inquiry, while community-level cognitive 
presence is the extent to which the entire group makes this progression. 

In a learning community, both faculty and other students support engagement, correction, deeper 
understanding, and learning. The selection and design of course activities creates more or fewer 
opportunities for higher-order thinking, resolution, and integration; for example, Kanuka and her 
colleagues found that debates and WebQuests produced higher levels of cognitive presence than did 
reflective deliberation, invited experts, or nominal group techniques [58], Both individual and group-level 
phenomena affect learning; Kanuka and Garrison identified three external processes that exist among 
members of a class that support cognitive presence—discourse, collaboration, and management—and 
three internal processes that a student must personally engage in—reflection, monitoring, and the 
construction of knowledge [84]. They concluded that “critical thinking is not solely a reflective, personal, 
and internal process. Rather, it is an iterative relationship between the personal and shared worlds” [84, p. 
25], 

Clearly, cognitive presence is also affected by individual differences in student motivation, the time 
students dedicate to their classwork, self-efficacy, and self-regulation [59, 71], There are also situational 
factors that support or hinder learning, particularly for adult students [76]. Students must participate, 
whether publicly or privately, in order to achieve deep learning; they must read resources and posts, ask 
and answer questions, reflect on ideas, explain concepts, evaluate options, and apply concepts to other 
situations. They must choose to take the time to read or listen to others’ contributions, and create their 
own. When teachers structure and support practical inquiry and deep learning, and students engage in 
discourse in support of learning, there will be collective-level cognitive presence. The more members of a 
learning community participate, the more a community is created in which deeper learning is possible and 
is shared among the members. As with social presence, cognitive presence is affected by students, 
teachers, and course design; therefore it should be examined at the level of the class section. 

Hypothesis 3 a: It is hypothesized that there will be strong agreement between members of a class section 
on perceptions of cognitive presence. 

Hypothesis 3b: It is hypothesized that class sections will differ in their community-level cognitive 
presence. 

IV. METHODOLOGY 


D. Sample 

The hypotheses were tested in a large private midwestem university. Respondents were 875 students 
enrolled in 126 sections of 44 unique courses in five different colleges. Classes were taught by 55 
different faculty over the course of three years, in three graduate programs in the Schools of Public 
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Service (5% of responses representing five class sections with three unique courses), Commerce (2.5% of 
responses, representing one class), and Education (8% of responses representing six class sections with 
two unique courses), and two undergraduate programs in Computer Science (1.8% of responses, with one 
class) and an interdisciplinary school for adult students (82.6% of responses, reflecting 113 sections with 
37 unique courses). The interdisciplinary courses included liberal arts, humanities, sciences, and 
professional studies. Most (85.5%) were undergraduate students. Students ranged from 19-69 years old, 
with an average age of 35.5 years old. Nearly two-thirds (64.7%) were female, and 35.3% were male. The 
vast majority (83.3%) had taken prior online courses, with a mean response of between four and five prior 
courses. 

E. Procedures and Measures 

Students were offered the opportunity to participate in the study by their instructors, some of whom 
offered extra credit for participation. Participants’ names were sent to researchers after classes had ended 
so that credit could be allocated. All classes but two were designed to create a community of inquiry, with 
the assistance of an instructional designer. All included active engagement in discussion, guided by the 
teachers; some involved weekly synchronous webinar meetings. Most were focused not only on 
developing competence but also on active application of knowledge to the real world. Courses taught 
more than once were “locked down” with prestructured syllabi, learning materials, activities, discussions, 
and assessments. Instructors were trained in the learning management system, and most were also trained 
in how to support active engagement in online courses. 

Faculty were invited to participate, and those who did sent information to their students. Some, although 
not all, offered extra credit for participation, which is a common practice in this university. Student 
participants were sent links to an online survey, which included the Community of Inquiry questionnaire 
[5, 6, 46, 69, 70, 85] along with questions about the technology used to teach the classes, and other 
outcomes, such as satisfaction. Only the community of inquiry responses are reported in this study. 
Response rates averaged 35.5%, which is consistent with response rates to online surveys [86]; class sizes 
ranged from 5-28 students, and the average number of participants per class was 6.9. The three scales in 
the survey had high internal consistency; the coefficient alpha was .95 for teaching presence, was .89 for 
social presence, and was .93 for cognitive presence. 

Following the recommendations of Liidtke et al. [87], LeBreton and Senter [34], and Biemann, Cole, and 
Voelpel [39] to include multiple indices of within-group agreement and between-group variance, multiple 
statistics are used to assess each. To test all of the hypotheses about the degree of agreement within 
groups (Hypotheses la, 2a, and 3a), three statistics are examined: r WG (j), 1CC(2), and AD M(V) . To assess 
the hypotheses about the variance among groups (Hypotheses lb, 2b, and 3b), two statistics are used: F- 
ratio and ICC(l). Only groups with three or more respondents were used. While some researchers have 
recommended using groups of 10 or more [88], Maloney and her colleagues conducted a Monte Carlo 
study of missing data in small group composition research; they found that both r WG(J) and ICC provide 
more accurate estimates of consensus and reliability when all groups are included, even those with only a 
single respondent [89]. However, r WG(J) can only be calculated on groups of two or larger, and the 
possibility of response bias from missing data/measurement error from a nonrepresentative sample led to 
a cutoff of three students per course. Given that the scales include between nine and fourteen items, and 
that there is a reciprocal relationship between the number of items and raters [34], inclusion of groups 
with three or more respondents was deemed appropriate. 

In order to calculate r WG(J) accurately, the distribution of the null hypothesis must be determined; this is 
the set of ratings that would be made by group members if there were no agreement whatsoever, raters 
were responding randomly, and the only factor influencing scores was response bias, such as central 
tendency, leniency, or severity. The likely bias must be established on theoretical grounds, considering 
the nature of the questions that are posed [34, 39]. 

The Community of Inquiry survey measures faculty behaviors in designing, developing, and leading 
online courses; student behaviors and beliefs about communication with one another to accomplish course 
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tasks; and student engagement in intellectual activities in the course. At least one large online university, 
American Public University, uses the Col instrument as its course evaluation. Therefore, one would 
expect the distribution of responses to reflect patterns of course evaluation responses, particularly for 
teaching presence. Prior research has found a slight positive skew in teaching evaluations. For example, 
one study of more than 60,000 Australian university responses found average ratings between 2.92 and 
3.17 on eight items about course design and instruction [90]; the response scale of 1-4 had a numerical 
mean of 2.5, and the average deviation from the mean was .61, or 15% higher. Another study of course 
evaluations in a California university found overall averages of 3.9 on a 1-5 scale, with an average 
deviation of .9 or 18% above the arithmetic mean of 3 [91]. We therefore hold the null hypothesis for 
teaching presence to reflect mild skew. 

The questions in the social and cognitive presence scales overlap with many aspects of the National 
Student Survey of Engagement (NSSE) [92], In particular, the NSSE "deep thinking" scale is comprised 
of three components: higher order thinking; integrative learning; and reflective learning. These include 
items that reflect working with others, and analyzing and applying information as found in the social and 
cognitive presence scales. One national study found that first-year online students averaged between 58, 
69, and 62 on these three scales, which were 8%, 12%, and 19% above the arithmetic mean, respectively 
[93]. Another national study found that seniors reported average scores of 69, 57, and 60 on the scales, 
which were 7%—19% above the arithmetic mean [94]. Items indicative of social presence (labeled as 
"curricular peer interaction") were found to be skewed for older students, with seniors reporting a range 
of 35%—71% and an average of 57%, which was 7% above the mean [95]. Overall, there is reason to 
assume a slight positive skew in the distributions of social and cognitive presence as well. 

Two programs were used to calculate statistics. First, SPSS syntax was created, following the description 
in LeBreton and Senter [34], to calculate the r WG(J) , the AD M( j), and the AD Me d(j> for every section. The 
r W G(j> statistics were calculated with a uniform null distribution as well as a slight skew. Second, the Excel 
tool developed by Biemann et al. [39] was used to calculate r WG(J) for each class as well as the 1CC(1), 
1CC(2) and F-ratios. This tool automatically replaces values lower than 0.0 and greater than 1.0 with 0.0, 
because such values reflect complete lack of consensus and, typically, a bimodal distribution [34], The 
distributions of a sample of such groups were plotted and examined, and all were found to be bimodal. 
The r WG(J) statistics of a sample of groups were compared across the two approaches to calculation and 
found to be identical for all indices greater than 0.0 and equal to or less than 1.0. 

V. RESULTS 


A. Social Presence 

Flypotheses la and lb addressed social presence. Flypothesis la held that there would be agreement 
among students in each class about the degree of social presence in the online community, and 
Flypothesis lb held that there would be differences between class sections. As described previously, the 
first of these hypotheses is tested with four analytical methods; Table 2 presents the first of these, r WG(J) . 
Perceptions of students in 126 class sections were examined, using both a uniform null distribution and a 
slightly skewed distribution. Following the procedure recommended by LeBreton and Senter [34], the 
proportion of groups with either very strong or strong agreement (>.70) are examined. Using a uniform 

null distribution, nearly all of the groups (124 out of 126 groups, or 98.4%) had either very strong or 

2 

strong agreement; using a slightly skewed null distribution ( cr £ = 1.34), the proportion is smaller but still 
substantial, with 81.0% (102) of the groups showing strong or very strong agreement. The averages are 
.93 (uniform null) and .82 (slightly skewed null), which are well above the 95% confidence levels of .61 
(uniform) and .74 (slightly skewed) reported by Cohen, Doveh, and Nahum-Shani for groups of 7 on 
scales with high internal consistency among items [96]. 
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Social presence 
agreement 

Size 

r WG(J) 

Uniform null 
distribution 

r WG(J) 

Slightly skewed null 
distribution 

very strong 

>.9 

103 

60 

strong 

.71—.9 

21 

42 

moderate 

.51-7 

2 

8 

weak 

.31-5 

0 

1 

none 

<.3 

0 

0 


out of range 

0 

8 


mean r wg<j) 

0.93 

0.82 


SD of mean r WG(J) 

0.06 

0.24 


Table 1: r WG(J) statistics for the social presence scale 


The second index, the AD M(J ),was 0.59, and the AD MeA{i) was 0.35. According to Burke and Dunlap [97], 
these are well below the cutoff of 0.80 for a 5-point scale and therefore indicate acceptable levels of 
agreement; 0.0 indicates no deviation, so a smaller score indicates greater within-group agreement. Both 
indices are also within the .05 significance level of the uniform distribution found by Cohen et al. for 
groups of seven on scales with high internal consistency reliability, (<0.99), and fall within the bounds for 
slightly skewed distributions (<0.70) [96, p. 159]. 

The 1CC(2) was 0.82; traditionally, this is assessed as a reliability coefficient, although it measures both 
consistency and consensus and therefore may be lower than indices that measure either consistency or 
consensus alone [34, 39]. In this case the result exceeds the minimum cutoff of 0.70. 

A large majority of class sections had strong agreement on r WG (j) falling within the range needed to meet 
the standard of p < .05 probability; the average levels fell within this standard as well, as did the 1CC(2) 
and the AD M(J ),with a uniform distribution. With all indices of agreement falling within cutoff levels at a 
probability ofp < .05, Hypothesis la is supported. 

Hypothesis lb holds that class sections differ on social presence. Analysis of variance produced an F-ratio 
of 5.44 (p < 0.000), and an 1CC(1) of 0.39, which is far above the minimum effect size of 0.05 [31, 36]. 
Hypothesis lb is also supported. 

B. Teaching Presence 

Hypotheses 2a and 2b addressed teaching presence, with Hypothesis 2a holding that there would be 
agreement among students in each class section about the degree of teaching presence, and Hypothesis 2b 
holding that there would be differences between class sections. Table 2 presents the first set of analyses 
for Hypothesis 2a, r WG(J) . With a uniform null distribution, 91.3% of sections (115 out of 126) had strong 
or very strong agreement, while 79.4% of sections (100) did so with a slightly skewed null distribution. 
The mean r WG(J) were .88 (uniform null) and .78 (slightly skewed null), both of which exceed the 95% 
confidence level minimums of .61 (uniform) and .74 (slightly skewed) [96]. 


Teaching presence 
agreement 

Size 

>'WG(J) 

Uniform null 
distribution 

>'WG(J) 

Slightly skewed null 
distribution 

very strong 

>.9 

103 

81 

strong 

.71-9 

12 

19 

moderate 

.51-7 

1 

0 

weak 

.31-5 

0 

2 

none 

<.3 

1 

3 


out of range 

8 

17 
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mean r wg<j) 

0.88 

0.78 


SD of mean r W G(j) 

0.25 

0.34 


Table 2: r WG (j) statistics for the teaching presence scale 

The second index, the was 0.62 for teaching presence, and the AD MtAW was 0.33. These fall below 

the 0.80 cutoff [95] as well as the estimates of Cohen et al. [96] for both uniform (<0.99) and skewed 
(<0.70) distributions. The 1CC(2) was 0.82, which well exceeds the minimum reliability requirement of 
0.70. 

With all indices of agreement meeting the minimum levels needed to reject the null hypothesis with a 
probability of p < .05, Hypothesis 2a is supported. 

Hypothesis 2b holds that class sections differ on their perceptions of teaching presence. The F-ratio was 
5.60 (p < .000), and the 1CC(1) was 0.40, far exceeding the minimum effect size of 0.05. Therefore, 
Hypothesis 2b is also supported. 

C. Cognitive Presence 

Hypothesis 3 a holds that there will be agreement within class sections on cognitive presence, and 
differences between class sections. Table 3 presents the results of the r WG(J) calculations. Using a uniform 
null distribution, 97.6% of all sections (123 out of 126) had either very strong or strong agreement, and 
87.3% of the sections (110) had very strong or strong agreement using a slightly skewed null distribution. 
The mean r w g<j) statistics were .94 (uniform) and .87 (slightly skewed), far exceeding the minimums of 
minimums of .61 (uniform) and .74 (slightly skewed) established by Cohen et al. [96]. 


Cognitive presence 
agreement 

Size 

>'WG(J) 

Uniform null 
distribution 

?'WG(J) 

Slightly skewed null 
distribution 

very strong 

>.9 

116 

84 

strong 

.71-9 

7 

26 

moderate 

.51-7 

1 

1 

weak 

.31-5 

0 

1 

none 

<.3 

0 

2 


out of range 

2 

6 


mean r wg<j) 

0.94 

0.87 


SD of mean r W G(j) 

0.13 

0.23 


Table 3: /'wg(J) statistics for the cognitive presence scale 


The AD M(I} was 0.54, and the AD Me dpi was 0.28, which demonstrate acceptable levels of agreement at the 
p < .05 level, according to both Burke and Dunlap (<0.80) [97] and Cohen et al. [96] for both uniform 
(<0.99) and skewed (<0.70) distributions. The 1CC(2) was 0.82, which again exceeds the 0.70 cutoff. 
Thus, all indices support Hypothesis 3 a. 

Hypothesis 3b holds that class sections differ on cognitive presence. This is supported by both the F-Ratio 
of 5.45 (p < .000) and the 1CC(1) of 0.39, which exceeds the 0.05 minimum. 

VI. DISCUSSION 

Fourteen years of research have found that online classes are more successful in supporting deep learning 
when they are characterized by a community of inquiry [1, 3, 4, 44, 74], An important advance was the 
development and validation of a survey instrument, which enables broader examination of the community 
of inquiry and factors that aid in its development [5, 6, 70, 85]. As this valuable instrument became 
widely used, attention focused on causal analysis and relationships among individual-level variables such 
as perceptions and student competences [46, 59, 71, 83, 98], While both individual and group-level 
behaviors and perceptions have been part of the Col framework, the question of level of analysis has not 
often been directly addressed. The concept of organizational climate provides a template for examining 
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levels in the Col, because both individual and group-level climate exist, and both levels affect one another 
reciprocally [11, 15, 29]. Both climates are typically measured with the same surveys, and perceptions of 
individuals may be aggregated if there is sufficient agreement among the members of a group to support 
the existence of a shared group perception [38, 88], 

This study posits that both individual and class-level Col are relevant to understanding how online 
learning occurs. The perceptions and actions of individual students and teacher(s) are critical; ultimately, 
every student experiences learning as a result of his or her actions. However, in a true community of 
inquiry, the actions of many members together create synergy and, if carefully guided, can lead to both 
trusting relationships and a deep level of learning [56, 58, 99, 100]. In this work, individual-level Col was 
formally distinguished from class, group, or community-level Col. 

This study examined agreement among perceptions of students in a class section on teaching, social, and 
cognitive presence. After describing the individual-level and group-level presences, it examined three 
indices of within-class agreement (r WG(J) ,4D M(J) , and 1CC(2)) and two indices of between-class 
differences (F-ratio and ICC(l)) on the Col survey instrument [19, 30, 34, 38, 88, 89, 96, 97, 101, 102, 
103], considering various possibilities for underlying response bias and examining deviation from means 
and medians. More than 125 classes were analyzed, and results strongly supported all hypotheses. All 
indices of agreement within class sections on social, teaching, and cognitive presences were significant, 
providing very strong evidence of community-level shared perceptions about Col presences. All indices 
of differentiation across multiple class sections were also significant, indicating that sections differed 
from one another in their teaching, social, and cognitive presences at the community level. 

Overall, the results found very strong support for the existence of a community of inquiry that is shared 
within each class section, and that differs across sections. The survey items that measure individual 
perceptions of teaching, social, and cognitive presences can also be used, when aggregated, to measure 
class-level teaching, social, and cognitive presences when agreement among students indicates that a 
learning community exists. This study provided statistical techniques to determine whether such a 
learning community exists, and evidence that it often does when built into the design of courses and 
expectations of teachers. It also provides evidence that the Col exists at multiple levels; individual 
students perceive their own actions, the actions of others, and the online class environment. To a large 
extent those perceptions are shared with others, creating a genuine community that exists among the 
members. 

This technique can be used by online teachers and administrators to examine the ability of teachers to 
create a strongly shared community within online classes. The individual levels of each presence, the 
average group levels, and the amount of agreement may be calculated to help instructors strengthen their 
teaching. For example, if low agreement indicates that subgroups have different levels of social presence 
in online courses, instructors may redesign courses to create group activities that engage all students, or 
modify grade distributions to reward interaction. Low agreement on cognitive presence may require 
redesigning learning activities or providing additional support for some students to enable them to fully 
integrate material. Low agreement about teaching presence may reflect that instructors are engaging more 
with some students than others, which can be interpreted as “playing favorites.” Examination of 
individual feedback would not readily reveal such issues, while examining agreement among class 
members would identify them easily. 

A few potential limitations of the study should be noted. Between 5% and 13% of groups had r WG( j) 
scores that were out of range, either because they were smaller than 0.0 or greater than 1.0, indicating 
bimodal distributions. Teaching presence had the most groups with bimodal distributions and out-of¬ 
range scores. It is possible that this reflects two subgroups within the class that may be produced by 
sampling error. A number of students participated because class credit was given, which may have 
created two subgroups of respondents: those who needed the credit and were not engaged with their 
teacher or the course, and those who felt closely connected with their teachers and wanted to honor their 
request. The r WG(J) for the out-of-range (bimodal) classes was set to 0.0, which may have increased the 
likelihood of a Type 2 error. However, the significant findings indicate that, if response bias occurred, it 
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did not substantively alter the 

results or conclusions. It would nonetheless be useful to examine agreement in situations in which all 
students complete the Col instrument to eliminate any possibility of response bias. 

This study is also limited by the fact that all data were collected in one university, and the overwhelming 
majority of the students were adults. In addition, the majority of the classes were taught in one school that 
adopts a social constructivist andragogy in its online courses. Most of these courses involve scaffolded 
learning, regular reflection on personal experience, real-world application, and intensive use of discussion 
for collaborative problem-solving [104, 105]. These approaches are likely to strengthen social and 
cognitive presences and enhance communal interactions, and therefore are likely to create stronger 
agreement among class members. These features increase the likelihood of a shared community of inquiry 
in the classes, but they are not representative of all online courses. It may be that in classes with more 
traditional pedagogies, focused on knowledge acquisition and multiple-choice testing, students interact 
differently with the teacher and content, creating less social presence and more variation among the 
students on teaching and cognitive presence. Further research in other contexts is needed to support the 
external validity of the findings. 

On the other hand, this study introduces a methodology for considering levels of analysis and aggregation 
in online courses. It may be applied to other concepts that exist at different levels, such as class or 
university leadership, pedagogy, and other factors. Climate itself might be addressed in online classes; 
whereas the organizational climate literature has examined the climate for safety, technological updating, 
and justice, the topics of interest in online learning might include the climate for academic integrity, for 
welcoming diverse ideas, or maximizing learning. 

While academic research has examined the relationships among individual responses to the Col presences 
[46, 56, 71, 83], the relationships among the group scores should also be examined. Both the average 
class level and the degree of agreement among class members may have important effects. For example, it 
may be that class-level social presence affects cognitive presence more strongly than individual-level 
social presence, which may explain some findings that have led to critiques of social presence [7, 8], The 
methodology introduced in this study could be used and extended to examine these possibilities. 

It is also possible that courses in different subject areas have different levels of agreement. Prior research 
has found differences in the levels of teaching, social, and cognitive presences across disciplines, with 
different levels in “hard” vs. “soft” disciplines as well as “pure” vs. “applied” ones [66, 68], As the 
pedagogies, student expectations, and instructor expectations all differ, both the level and agreement 
about the Col will likely differ as well. 

Another potentially fruitful area to explore is the effect of the strength of the agreement among members 
on outcomes. There is some evidence that the strength of agreement on perceptions about climate affects 
group outcomes; it is not just the average level, but how closely everyone agrees with one another that 
determines what groups do, because greater consensus creates a stronger force on individuals [17, 39, 
106]. The same may occur in online classes; future research should examine the effect of class agreement 
on outcomes such as learning or satisfaction. 

In sum, the analytical approach presented here enables practitioners and researchers to measure both 
individual and class-level teaching presence, social presence, and cognitive presence quickly and easily, 
using the validated and widely used Col survey. This may help improve course instruction, facilitate 
research at the group level, and clarify conceptual issues regarding the role of community in online 
courses. 
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VIII. TECHNICAL APPENDIX 

The most popular statistic for measuring organizational climate was preferred by James, Demaree, and 
Wolf [38], who developed r WG , in which within-group variance is divided by a hypothetical variance that 
would result if there were no consensus among raters (the null hypothesis), and therefore all variance 
were due to measurement error. The formula for r WG(J) is: 
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where J is the number of items in a scale, n is the mean observed variance for J items, and <3+ is the 

variance expected when there is a complete lack of agreement, or error variance. In some situations, this 
hypothetical error variance may be uniform across all response options; that is, evaluators using a 1-5 
scale are as likely to select the number 1 as they are 2, 3, 4, and 5. For scales with multiple items, James, 
Demaree, and Wolf recommended the r WG(J ) statistic, which combines the variances of each item in the 
scale. If there is reason to believe that systematic response bias, such as leniency or severity error, would 
affect ratings, then the hypothetical error variance would differ from a uniform distribution, and can be 
calculated. They calculated null variances to use for common skewed distributions for multi-item 
measures, and LeBreton and Senter added additional variances for a wider range of scale types [34]. 

This statistical index is widely favored because it has an easily interpretable range, from 0 to +1.0, where 
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0 indicates no agreement between raters and 1.0 indicates perfect agreement. This index measures true 
consensus, or agreement both on a relative and absolute basis, rather than consistency or reliability, which 
refer to agreement only on a relative basis [34, 87, 88, 107]. For example, if one group member responds 
to six items with scores of 5, 4, 5, 4, 4, and another responds to the same items with scores of 2, 1,2, 1, 1, 
there is complete consistency in their use of the scale; therefore, the inter-rater reliability as measured by 
correlation would be 1.0. However, the ratings are nearly opposites, and there is little actual consensus 
between the raters; the r WG(J) would be very low. 

Despite its popularity, this statistic has been critiqued on several grounds, including the notion that it does 
not assess inter-rater reliability as claimed [108]. It also requires that the hypothetical null distribution be 
specified, based on knowledge of likely survey response bias [34, 38, 103]. In addition, it can produce 
uninterpretable numbers below 0.0 and above 1.0 when the observed variance is larger than the 
hypothetical null distribution; this is particularly likely if a strong skew is used for the null distribution, if 
the scores are bimodal, or if there are many items in a scale [27, 107]. Adding items to a scale increases 
the internal consistency of r WG( l) because it corrects for attenuation [34]. 

The recommended way to deal with uninterpretable values—greater than 1.0 or less than 0.0—is to 
replace them with the value 0.0, which indicates no consensus among the group members [34, 38, 39], In 
an elegant simulation, Roberson and her colleagues found that nearly all indices of consensus or 
dispersion are strongly intercorrelated for single-item measures, with correlations between .93 and .98 
[103], 

Many researchers have used r WG or r WG(J) to determine whether perceptions of different people about 
organizational climate are similar enough to aggregate [14, 17, 21, 22, 23, 32, 36, 37]. Some provide 
analyses using multiple comparison null distributions, including both uniform and skewed distributions; 
skewed distributions tend to produce lower r WG(J) scores, but are conceptually more accurate because 
response bias is such a common phenomenon with survey data [34], There is no clear agreement on the 
optimal distribution for a null hypothesis, nor a smallest sample size that should be used to assess a group 
score, nor a means of dealing with missing data. Recent analyses have recommended a graduated standard 
for acceptable levels, rather than a uniform cutoff: .00-30 = lack of agreement; .31—.50 = weak 
agreement; .51—.70 = moderate agreement; .71-90 = strong agreement; and .91-1.00 very strong 
agreement [34, 39], 

Another means of assessing agreement is average deviation (AD), which is the absolute value of the 
difference between each item and the mean or the median; either can be calculated [34, 103], Because this 
index is not squared, it is not unduly sensitive to outliers. However, it reflects the numbering in the scale, 
and measures dispersion so larger numbers indicate more disagreement within groups [103], The formula 
for AD is: 


A 



where X is the score on item j, X J is the average score on item j, K is the number of judges, and k= 1 to 
A judges. AD M( j) is the mean of the AD M( j > of all items in the scale. The AD^O) has been used to measure 
climate strength [103] and to assess the reliability of climate perceptions, both in organizations [14] and 
in schools [87]. The 4/J V icd(.i) is calculated based on medians rather than means, and is more robust as it is 
less influenced by outliers [34, 109]. 

In addition to measuring consensus, research on organizational climate also assesses the relative 
consistency of perceptions by examining the overall within-group and between-group variation [34]. Two 
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common indices are based on analysis of variance: intraclass correlations 1 (ICC(I)) and 2 (1CC(2)). The 
1CC(1) measures the proportion of overall variance that reflects between-group differences; it can be 
considered the effect size attributable to the group alone, or the extent to which ratings are affected by the 
group to which one belongs [34], The formula is: 

/ecu)--r-r 

r +o 

where f is the variance between groups, and (T is the variance within groups [87, 110], Values of .10 to 
.24 are considered medium, and of .25 or higher are considered large; even small numbers can indicate 
that group membership affects scores [34]. 

1CC(2) measures the reliability of group mean scores [87, 110], with high scores indicating that mean 
(aggregated) scores of the groups are relatively free from variation among the members [111]. The 
formula is: 


ICC( 2) = 


K x 7CC(1) 
i + (A:-i)x/cc(i) 


where K is the number of judges; the average is used if groups have different sizes. A graduated set of 
reliability standards is recommended for evaluating it, although the traditional reliability cutoff of .70 is 
often used [34], Several scholars have used the 1CC(1) and 1CC(2) to validate the presence of group-level 
climate in organization [14, 17, 22, 23, 30, 36]; their use has been recommended as a matter of routine in 
research on classroom agreement [87]. 

Others use the simple F-statistic in one-way analysis of variance to determine whether individual climate 
perceptions can be aggregated into group-level measures, which compares between-group variance with 
within-group variance. This statistic may be problematic if there is restriction of range or when comparing 
groups across different organizations [27, 38], It has frequently been used to provide evidence of 
between-group differences on climate, indicating that climate is specific to each group [14, 22, 23, 26, 30, 
32,37], 
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